System and method for interfacing between a display and a controller

ABSTRACT

A system includes a first device configured to cause a virtual reality image to be output by a display. The system also includes a second device communicatively coupled with the first device. The second device is configured to detect a user input associated with an action capable of being displayed in the virtual reality image. The second device is also configured to cause the first device to cause an object to be displayed in the virtual reality image based on the user input. The object at least partially includes a graphic representative of one or more of the second device, a user movement with respect to the second device, or the user input.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional ApplicationNo. 62/288,388, filed Jan. 28, 2016, the disclosure of which is herebyincorporated by reference herein in its entirety.

BACKGROUND

Users sometimes interact with software by way of a controller configuredto receive a user input. The controller is often a tactile device thatis held or touched by a user. Users sometimes view an image associatedwith the software via a display. Users usually provide a user input tothe controller while viewing the display. Device manufacturers andservice providers are continually challenged to enhance a user'sinteraction with a virtual reality image output by a display.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isnoted that, in accordance with the standard practice in the industry,various features are not drawn to scale. In fact, the dimensions of thevarious features may be arbitrarily increased or reduced for clarity ofdiscussion.

FIG. 1 is a block diagram of a system, in accordance with someembodiments.

FIG. 2 is a block diagram of a controller, in accordance with someembodiments.

FIG. 3 is a block diagram of a main body device, in accordance with someembodiments.

FIG. 4 is a user interface generated based on a swipe operationperformed by a user, in accordance with some embodiments.

FIG. 5A is a user interface generated based on a tap operation performedby the user, in accordance with some embodiments.

FIG. 5B is a user interface generated based on a tap operation performedby the user, in accordance with some embodiments.

FIG. 6 is a user interface generated based on a tap operation performedby the user, in accordance with some embodiments.

FIG. 7A is a screen to be displayed on the HMD, which is generated basedon a pinch-in/pinch-out operation performed by the user, in accordancewith some embodiments.

FIG. 7B is a user interface generated based on a pinch-in/pinch-outoperation performed by the user, in accordance with some embodiments.

FIG. 7C is a user interface generated based on a pinch-in/pinch-outoperation performed by the user, in accordance with some embodiments.

FIG. 8A is a user interface generated based on motion input obtainedfrom an accelerometer or a gyroscope, in accordance with someembodiments.

FIG. 8B is a user interface generated based on motion input obtainedfrom the accelerometer or the gyroscope, in accordance with someembodiments.

FIG. 8C is a user interface generated based on voice input obtained fromthe accelerometer or the gyroscope, in accordance with some embodiments.

FIG. 9A is a user interface generated based on motion input obtainedfrom the accelerometer or the gyroscope, in accordance with someembodiments.

FIG. 9B is a user interface generated based on motion input obtainedfrom the accelerometer or the gyroscope, in accordance with someembodiments.

FIG. 9C is a user interface generated based on motion input obtainedfrom the accelerometer or the gyroscope, in accordance with someembodiments.

FIG. 10A is a user interface generated based on voice input obtainedfrom a microphone sensor, in accordance with some embodiments.

FIG. 10B is a user interface generated based on an input obtained fromthe microphone sensor, in accordance with some embodiments.

FIG. 11 is a flow chart of a process to be performed by the controller,in accordance with some embodiments.

FIG. 12 is a flow chart of a process to be performed by the main bodydevice, in accordance with some embodiments.

FIG. 13 is a flow chart of a process of connecting the controller andthe main body device, in accordance with some embodiments.

FIG. 14 is a block diagram of a system comprising a plurality ofcontrollers, in accordance with some embodiments.

FIG. 15 is an user interface in which one virtual reality image isaccessed by a plurality of users by way of a plurality of controllers,in accordance with some embodiments.

DETAILED DESCRIPTION

The following disclosure provides many different embodiments, orexamples, for implementing different features of the provided subjectmatter. Specific examples of components and arrangements are describedbelow to simplify the present disclosure. These are, of course, merelyexamples and are not intended to be limiting. For example, the formationof a first feature over or on a second feature in the description thatfollows may include embodiments in which the first and second featuresare formed in direct contact, and may also include embodiments in whichadditional features may be formed between the first and second features,such that the first and second features may not be in direct contact. Inaddition, the present disclosure may repeat reference numerals and/orletters in the various examples. This repetition is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed.

A head-mounted display (HMD) is a display that is wearable on a user'shead. An HMD is capable of presenting images to the user with a displayor the like arranged right in front of the user's eyes. Some HMD's areconnected to an external network, e.g., the Internet, via a main bodydevice. Conventional controllers by which a user is able to interactwith software that is associated with an image presented by way of theHMD do not communicate to or from the external network. Additionally,conventional controllers are often a dedicated controller for connectionto a HMD. Sometimes, when a dedicated controller is necessary foroperation of the HMD, the user is required to carry the dedicatedcontroller for every HMD operation.

Some conventional controllers that are usable to interact with softwarethat is associated with an image presented by way of the HMD ofteninclude buttons, keys or motion sensors to facilitate performance of oneor more operations or interactions. Such controllers however, do notinclude a sensor display, which makes performance of some intuitive andfiner operations difficult.

For example, if a controller is configured to cause one or moreoperations to occur via software associated with an image presented byway of the HMD based on a user's actuation of a button or key, and auser is completely immersed in a virtual space while wearing the HMD,the user often has difficulty recognizing a current state of thecontroller that the user is attempting to operate. For example, a usermay have difficulty understanding which key or button the user's fingeris placed. As such, when the user's finger is displaced from a desiredkey or button, the user may perform an undesired operation.

One operation screen is sometimes shared and operated by a plurality ofusers located in a same space, and a plurality of controllers areconnected to one main body device. A multi-controller configurationoften increases demand for system resources and causes a high load to beplaced on the main body device. Additionally, when one operation screenis shared and operated by a plurality of users located in differentspaces, the plurality of controllers are connected to different mainbody devices in the respective spaces. Thus, communication is made byconnecting each of the main body devices to a server via a network. Inthis situation, different main body devices are necessary for theplurality of controllers to facilitate user interaction with thesoftware associated with the image presented by way of the HMD.

FIG. 1 is a block diagram of a system in accordance with someembodiments.

The system includes a controller 100 and a main body device 120connected to the controller 100 by way of a network 180. In someembodiments, the network 180 is wired network. In some embodiments, thenetwork 180 is a wireless network. In some embodiments, network 180comprises a combination of wireless and wired network connections. Thesystem also includes a head-mounted display 140 connected to the mainbody device 120 by way of a network 200. In some embodiments, network200 is a wired network. In some embodiments, network 200 is a wirelessnetwork. In some embodiments, network 200 comprises a combination ofwired and wireless network connections.

By way of example, one or more of network 180 or network 200 include oneor more networks such as a wired data network, a wireless network, atelephony network, or a combination thereof. In some embodiments, one ormore of network 180 or network 200 comprises any local area network(LAN), metropolitan area network (MAN), wide area network (WAN), apublic data network (e.g., the Internet), short range wireless network,a packet-switched network, such as a commercially owned, proprietarypacket-switched network, e.g., a proprietary cable or fiber-opticnetwork, or any other suitable packet-switched network, some othersuitable data network, or any combination thereof. In some embodiments,a wireless network comprises a cellular network and may employ varioustechnologies including enhanced data rates for global evolution (EDGE),general packet radio service (GPRS), global system for mobilecommunications (GSM), Internet protocol multimedia subsystem (IMS),universal mobile telecommunications system (UMTS), etc., as well as anyother suitable wireless medium, e.g., worldwide interoperability formicrowave access (WiMAX), Long Term Evolution (LTE) networks, codedivision multiple access (CDMA), wideband code division multiple access(WCDMA), wireless fidelity (WiFi), WiGig, wireless LAN (WLAN),Bluetooth®, Internet Protocol (IP) data casting, satellite, mobilead-hoc network (MANET), other suitable communication network or system,or any combination thereof.

The controller 100 includes an antenna 110, and is thus capable ofconnecting to a network, e.g., the Internet, to communicate to/from aserver 160 without the main body device 120. The server 160 comprises aprocessor, one or more data storage devices, and a data communicationdevice, and is configured to transmit or receive data to or from thecontroller 100. In some embodiments, the server 160 is configured tomanage communications between a plurality of controllers 100.

The controller 100 includes at least one processor 102, at least oneinterface 104, at least one memory 106, at least one sensor display 108,and the antenna 110. In some embodiments, controller 100 is a computingdevice or system that is any type of mobile terminal, fixed terminal, orportable terminal including a mobile handset, station, unit, device,multimedia computer, multimedia tablet, Internet node, network node,satellite, communicator, desktop computer, laptop computer, notebookcomputer, netbook computer, tablet computer, personal communicationsystem (PCS) device, personal navigation device, personal digitalassistants (PDAs), audio/video player, digital camera/camcorder,positioning device, television receiver, radio broadcast receiver,electronic book device, game device, or any combination thereof,including the accessories and peripherals of these devices, or anycombination thereof. In some embodiments, the control 100 is or cansupport any type of interface to the user (such as “wearable” circuitry,etc.). For ease of discussion, the controller 100 is generally referredto as a mobile device that is held or worn by a user. In someembodiments, the controller 100 is configured to detect one or moretypes of operation input based on the user's operation, user motioninput based on the user's motion, voice input, or other suitable inputfor interacting with software associated with the image presented by wayof the HMD.

The controller-side processor 102 is configured to perform various typesof processing such as execution of a controller-side interface program106A and instruction of commands to respective functional units. Thecontroller-side interface program 106A is usable to achieve functions ofa transmission unit and a reception unit (FIG. 2) of the user controllerto be described later.

The controller-side interface program 106A is usable to achieve afunction of a transmission unit 204 to be described later, to therebytransmit output data from the controller 100 to the main body device 120via the network 180. Further, the controller-side interface program 106Ais used to achieve a function of a reception unit 206 to be describedlater, to thereby receive, from the main body device 120 to thecontroller 100, various types of data including experience information,e.g., an image that the user experiences in a virtual space.

The controller-side interface 104 is an interface configured to transmitoutput data to the main body device 120 via the network 180 and toreceive data from the main body device 120. The controller-sideinterface 104 is achieved by implementing the controller-side interfaceprogram 106A. In more detail, the controller-side interface 104 is aninterface configured to transmit, to the main body device, data obtainedfrom various sensors built into the controller, or output data, e.g., animage stored in the memory of the controller, and/or to receive, fromthe main body device, experience information such as an image or amoving image that the user experiences in a virtual space when the userwears a head-mounted device and is immersed in the virtual space.

The controller-side memory 106 stores the controller-side interfaceprogram 106A. Further, the controller-side memory 106 can store varioustypes of data such as images and moving images taken by the user in thepast and note information stored in the past, and data obtained fromvarious sensors, which is generated based on operations and motionsperformed by the user.

The sensor display 108 is a touch sensor configured to detect one ormore of a swipe operation, a pinch-in/pinch-out operation, a tapoperation, or some other suitable contact-based operation performed bythe user. The sensor display 108 enables a user to perform intuitive andfine operations through the swipe or pinch-in/pinch-out operation, forexample. The sensor display 108 makes it possible for a user to performan intuitive operation by way of the swipe operation, the tap operation,the pinch operation, or some other suitable contact-based input withoutconfirming the positions of the keys and buttons.

The controller 100 includes the antenna 110, and the controller 100 isitself configured to communicate to/from an external network, e.g., theInternet. In some embodiments, the controller 100 is configured toacquire in real time various types of information that can be acquiredvia the external network, such as weather information, map information,and applications, without applying a load to the main body device duringexecution of the application, to thereby reflect the acquiredinformation to an application executed by the main body device or adisplay screen of the HMD. In some embodiments, the controller isconfigured to download the controller-side interface program 106A fromthe server 160 to the user controller 100 via the network, e.g., theInternet, by the antenna 110, to thereby install the controller-sideinterface program 106A to the user controller. When the user installsthe controller-side interface program 106A to the controller 100, thecontroller 100 can be connected to the main body device 120 with use ofan arbitrary controller, e.g., a smart phone, without using a controllerdedicated to a head-mounted device. In this manner, the output data fromthe controller 100 can be reflected to the display of the head-mounteddevice 140.

In some embodiments, the controller 100 includes one or more of amicrophone configured to detect a voice input from the user, anaccelerometer a gyroscope configured to detect a motion input performedby the user, buttons, keys, a camera, a GPS unit, or a vibration device(not shown).

The microphone is configured to detect the voice or the like of the userholding or wearing the controller 100. Including the microphone enablesreflection of the words spoken by the user to a display object to bedisplayed on the HMD 140. Further, with use of a voice recognitionapplication stored in the memory, the words spoken by the user can beaudibly recognized so that, for example, on a game application executedby the main body device 120, the display object can perform motions,e.g., opening a door when the user speaks the magic words.

The accelerometer and the gyroscope are configured to detect motions ofthe user holding or wearing the controller 100, e.g., a user's motion ofswinging around the controller or throwing an object. Further, theaccelerometer is configured to detect the number of steps taken by theuser. Further, including the accelerometer, the gyroscope, and thesensor display 108 enables instantaneous conversion of a motion of thedisplay object (jumping of the display object), which corresponds to oneoperation input to the controller 100 (for example, tap operation), intoanother motion of the display object (sliding of the display object)with easy switching operations, e.g., shaking the terminal. Inparticular, the tap operation is an easy operation for the user wearingthe HMD, and hence various motions of the display object, such asshooting, cutting with a sword, and jumping, are allocated to the tapoperation on the application. When the user is immersed in the virtualspace while wearing the HMD, the user cannot directly visually refer tothe controller 100 being operated. With use of the accelerometer and thegyroscope, the user can instantaneously convert the motion of thedisplay object reliably with easy and intuitive operations, e.g.,shaking the terminal.

The buttons and the keys receive the operation input performed by theuser holding or wearing the controller 100.

The camera is configured to take pictures and moving images. In someembodiments, the taken pictures and moving images are stored into thememory of the controller 100. in some embodiments, images and movingimages captured by the camera and/or stored in the memory of thecontroller 100 can be reflected to the virtual space displayed on thehead-mounted display.

The GPS unit is configured to detect a position of the user holding orwearing the controller 100. Therefore, for example, a display screen ofthe HMD 140 can be generated based on the detected position of the user.For example, when the user is located in Hokkaido, a character of thedisplay object to be operated by the user in the virtual space displayedon the HMD can be set to a Hokkaido-specific character.

The vibration device is configured to vibrate the controller 100 inassociation with a game when, for example, the user wearing the HMD 140and playing a game is attacked in the virtual space displayed on the HMD140. The information experienced by the user during execution of theapplication in the virtual space is transmitted by a main bodydevice-side interface program 126A to the interface program 106A via thenetwork 180. Then, the vibration device vibrates the controller 100based on the information experienced by the user, which is received bythe controller-side interface program 106A. Vibration of the controller100 enables further immersion of the user into the virtual space.

The main body device 120 is configured to transmit or receive data to orfrom each of the user controller 100 and the head-mounted display 140.In some embodiments, the main body device 120 includes at least oneprocessor 122, at least one interface 124, and at least one memory 126.

The main body device-side processor 122 is configured to perform varioustypes of processing such as instruction of commands to respectivefunctional units, execution of various applications, e.g., a game,execution of the main body device-side interface program 126A, executionof an image generation program 126B for performing processing ofgenerating and outputting an image, and sound generation processing.

The main body device-side interface 124 is an input/output interfaceconfigured to receive output data transmitted from the controller 100via the network, transmit predetermined data from the main body device120 via the network, and output, to the HMD 140, a generated image inwhich the display object is arranged.

The main body device-side memory 126 is configured to store the mainbody device-side interface program 126A, the image generation program126B, and programs and data for achieving various functions to beexecuted by the processor 122 of the main body device 120.

The main body device-side interface program 126A is usable to achieve afunction of an input detection unit 304 (FIG. 3) to be described later.The main body device 120 is configured to detect and receive the outputdata transmitted from the controller 100, and to transmit various typesof data from the main body device 120 to the controller 100 by executingthe main body device-side interface program 126A. The main bodydevice-side interface program 126A is installed in the main body device120. For example, the main body device-side interface program 126A canbe downloaded to the main body device 120 via a network (not shown),e.g., the Internet or Intranet connected to the main body device 120, orcan be installed with use of a recording media. By installing the mainbody device-side interface program 126A in the main body device 120, andinstalling the controller-side interface program 106A to be describedlater in the controller 100, the user can use an arbitrary controller,e.g., a mobile terminal or a smart phone, instead of using a controllerdedicated to the HMD, to thereby reflect the user input from thecontroller 100 or the like to the HMD 140, or transmit, to thecontroller 100, the display on the HMD 140 and information of theapplication being operated. Further, the main body device-side interfaceprogram 126A makes it possible to acquire and identify output dataobtained from each of a plurality of controllers 100, to thereby connectthe plurality of controllers 100 to a single main body device 120.

The image generation program 126B is usable to achieve functions of anoperation determination unit 302, a motion determination unit 306, adisplay calculation unit 308, and an image generation/output unit 310 tobe described later.

The HMD 140 is connected to the main body device 120, and is configuredto display a generated image output from the main body device 120. Insome embodiments, the HMD 140 is configured to virtually display thesensor display (reference symbol 602 in FIG. 6), the buttons, and thekeys of the controller 100 held or worn by the user.

FIG. 2 is a block diagram of the controller 100, in accordance with someembodiments. The controller 100 includes an operation unit 202, atransmission unit 204, a reception unit 206, a communication unit 208,and an image pickup unit 210. The operation unit 202, transmission unit204, reception unit 206, communication unit 208, and image pickup unit210, or a part of any function thereof, is achieved by executing apredetermined program stored in the memory 106 of the user controller100 by the processor 102. The transmission unit 204 and the receptionunit 206 of the controller 100 are achieved by executing thecontroller-side interface program 106A by the controller-side processor102. The interface program 106A is downloaded from the server 160 viathe Internet with use of the antenna 130 of the controller 100, and isthus installed in the controller 100.

The operation unit 202 is configured to acquire, from one or moresensors included in the controller 100, one or more inputs relating topredetermined motions or operations performed by the user to thecontroller 100. The one or more sensors included in the controllercomprise one or more of a display sensor, an accelerometer, a gyroscope,a microphone sensor, a GPS sensor, or other suitable sensor. In someembodiments, the processor 102 of the controller 100 is configured tocombine one or more inputs detected based on the one or more sensorswith one or more inputs from other operation units such as buttons andkeys (for example, operation input of performing a motion of swingingthe controller while pressing the button of the controller). In someembodiments, when the user moves the controller 100, the operation unit202 is configured to acquire, from the accelerometer and the gyroscope,acceleration and an angular velocity with respect to the motion. In someembodiments, when the user performs a swipe operation, apinch-in/pinch-out operation, or a tap operation on the sensor display108 of the controller 100, the operation unit 202 is configured toacquire, from the sensor display 108, the types of the operations and/orvarious types of data relating to those operations, such as contact tothe display (tap), contact time (long press/tap), the number of times ofcontact within a predetermined time period (successive taps, singlepress), the displacement amount of continuous contact on the display(swipe distance, pinch distance), the contact displacement direction(swipe direction, pinch direction), and the contact displacement speed(swipe speed, pinch speed). in some embodiments, when the user speaks tothe microphone of the controller 100, the operation unit 202 isconfigured to acquire the user voice input from the microphone. In someembodiments, the operation unit 202 is configured to store those piecesof information such as the acceleration or angular velocity, operationinput information, voice input information, and/or position informationto the memory.

In some embodiments, the acceleration and the angular velocity obtainedfrom the accelerometer and the gyroscope are collectively referred to as“motion input information.” Further, the contact to the sensor display,the displacement amount of the continuous contact to the display, thecontact displacement direction, and the contact displacement speed,which are obtained from the sensor display when the user performs theswipe operation, the tap operation, or the pinch operation on the sensordisplay, are collectively referred to as “operation input information.”

The transmission unit 204 is configured to transmit, as output data,those pieces of information stored in the memory to the input detectionunit 304 via the network 180 in response to a request from the main bodydevice 120. The output data corresponds to the motion input information,the operation input information, the voice input information, and theposition information, which are output from the operation unit 202 andrelate to the predetermined motions and operations performed to thecontroller 100, and images and moving images stored in the memory 106 ofthe controller 100, or combinations thereof.

The reception unit 206 is configured to receive information transmittedfrom the main body device 120. If the user is immersed in the virtualspace while wearing the HMD 140, the reception unit 206 is configured toreceive information experienced by the user in the virtual space. Forexample, when the user playing a game is attacked in the virtual spacedisplayed on the HMD 140, the reception unit 206 receives theinformation of being attacked. Then, the controller-side processor 102executes a vibration function based on the received information of beingattacked, to thereby vibrate the controller 100. In some embodiments,reception unit 206 is configured to receive videos displayed whileplaying a game from the main body device 120. The controller-sideprocessor 102 is configured to store the received videos to the memoryof the controller 100, or display the videos on the controller-sidedisplay 108.

The communication unit 208 is configured to download various types ofdata, e.g., a program, from the server 160, upload videos being operatedby the user to the server 160, and directly communicate to/from anothercontroller 100 if there are a plurality of controllers 100. In general,the communication unit 208 performs communication via the Internet. Insome embodiments, the communication unit 208 enables the controller 100to connect to the external server 160 without a connection to the mainbody device 120. Therefore, the controller 100 can install or update thecontroller-side interface program 106A in an arbitrary location evenwithout connection to the main body device 120 as long as the controller100 is in an environment connectable to the Internet. Further, thecontroller 100 can acquire various types of information that can beacquired from the Internet (weather information, stock priceinformation, and map information), to thereby reflect the information tothe screen being operated by the user or the content of the game beingoperated by the user. In some embodiments, if there are a plurality ofcontrollers 100, the controller 100 is configured to communicate to/fromanother controller 100 via the communication unit 208 without the mainbody device 120.

The image pick-up unit 210 is configured to capture images or movingimages in a real space. Including the image pick-up unit 210 makes itpossible for images or the moving images taken in advance by the user tobe output to a display being operated by the user or a display of thegame being played.

The GPS unit is configured to provide position information of thecontroller 100 acquired by the GPS unit to the display being operated bythe user. For example, when the controller 100 is located in Hokkaido, acharacter of the display object to be operated by the user in thevirtual space displayed on the HMD 140 can be set to a Hokkaido-specificcharacter, or the user operation screen of the virtual space displayedon the HMD 140 can have a background of Hokkaido.

FIG. 3 is a block diagram of the main body device 120, in accordancewith some embodiments. The main body device 120 includes an operationdetermination unit 302, an input detection unit 304, a motiondetermination unit 306, a display calculation unit 308, and an imagegeneration/output unit 310. The operation determination unit 302, inputdetection unit 304, motion determination unit 306, display calculationunit 308, image generation/output unit 310, or a part of any functionthereof, is achieved by executing, by the processor 122, a predeterminedprogram stored in the memory of the main body device 120.

The operation determination unit 302 is configured to first determine,based on the content of the application being executed in the main bodydevice 120, what user operation input is required by the application.The required user operation is determined based on, for example, thecontent of the application being executed and a situation of the currentuser operation in the application being executed, and refers to a useroperation with respect to the controller 100, which is allowed by theapplication being executed to be input. For example, when the user playsa battle game in a virtual space, examples of the required useroperation that corresponds to an operation state during battle withenemies include “punching”, “shooting”, and “fending off an attack”.

The input detection unit 304 is configured to acquire, via the network180, output data corresponding to the required user operation among thepieces of output data stored in the memory of the controller 100. Forexample, when the required user operation is “punching,” the output datacorresponding to the required user operation is operation inputinformation (contact to the display by the tap operation). In someembodiments, output data can include the number of times of contactwithin a predetermined time period. The input detection unit 304 isconfigured to output the detected output data to the motiondetermination unit 306. In some embodiments, if the required useroperation is “punching” in the application being executed, the inputdetection unit 304 detects and acquires only the contact to the display,which corresponds to the tap operation, among the pieces of operationinput information of the output data. The motion determination unit 306can ignore the output data not corresponding to the required useroperation, e.g., a voice input when the required user operation is“punching.”

The input detection unit 304 is configured to detect, as the output datafrom the controller 100, data corresponding to the required useroperation determined by the operation determination unit 302, amongvarious types of information transmitted from the transmission unit 204of the controller 100 via the network 180, e.g., the user's motion inputinformation (acceleration and angular velocity obtained from theaccelerometer and the gyroscope), the operation input information (typessuch as swipe operation and pinch-in/pinch-out operation, and/or contactto the sensor display, displacement amount of continuous contact to thedisplay, contact displacement direction, and contact displacementspeed), voice input information (voice and volume obtained from themicrophone sensor), the images, the moving images, and the positioninformation, or combinations thereof.

In some embodiments, the input detection unit 304 is configured todetect the output data during a predetermined time period at apredetermined timing. In some embodiments, the input detection unit 304is configured to change the intervals of the timing to receive theoutput data. For example, if the display object is caused to performrough operations, e.g., turning a knob of a door in a virtual space, theintervals of the timing for detection can be increased to detect theinput 15 times per second. If, in the virtual space, the display objectis caused to perform motions that require sequentially reflecting, tothe screen, fine motions of a steering wheel, e.g., driving anautomobile, the intervals of the timing for detection can be decreasedto detect the input 60 times per second.

Table 1 includes example relationships among a content of an applicationbeing executed, a required user operation corresponding to the contentof the application being executed, and output data corresponding to therequired user operation. The content of the application, the requireduser operation, and the output data shown in Table 1 are merelyexamples, and a person skilled in the art may easily understand thatother user operation states and other user operations may be included.

TABLE 1 Content of application Required user Output data correspondingto required being executed operation user operation Battle Shooting/Operation input information (contact to punching display by tapoperation) Fending off Motion input information (acceleration, angularvelocity) Sliding Operation input information (contact to display by tapoperation) Swinging Motion input information (acceleration, aroundangular velocity) Fishing Casting/ Motion input information(acceleration, tugging angular velocity) Darts Throwing Motion inputinformation (acceleration, angular velocity) Image Scrolling Operationinput information (direction viewing of swipe operation, displacementamount, displacement speed), and images/moving images Zooming-in/Operation input information (direction zooming-out of pinch operation,displacement amount, displacement speed), and images/moving imagesVoice/image Speaking Voice input information (voice, volume) conversionout loud Driving Operating Motion input information (acceleration,steering wheel angular velocity) Accelerating Button/key inputinformation Stopping Button/key input information Tearing apart/ Tearingapart/ Operation input information (direction bringing bringing ofpinch/swipe operation, displacement together together amount,displacement speed) Zooming-in/ Zooming-in/ Operation input information(direction zooming-out zooming-out of pinch operation, displacementamount, displacement speed) Map viewing, Position data informationlandscape rendering, character rendering

Next, the motion determination unit 306 determines the motion to beperformed by the display object displayed on the HMD 140, based on theoutput data detected by the input detection unit 304 in accordance withthe required user operation. For example, if the required user operationis “punching,” the motion to be performed by the display object in thevirtual space (punching, frequency of punching) is determined dependingon the sense of contact to the display and the number of times ofcontact, which are included in the output data. The display objectrefers to an object to be operated by the user in a virtual space, andexamples of the display object include, when the user is playing a game,game characters and game items, and when the user displays taken imagesor moving images, those images and moving images.

In some embodiments, if the required user operation is “scrolling animage,” the motion determination unit 306 is configured to acquire,among the pieces of operation input information of the output data, thedirection of the swipe operation, the displacement amount, and thedisplacement speed. Then, the motion determination unit 306 isconfigured to determine the motion to be performed by the display object(scrolling an image) based on the direction of the swipe operation tothe display, the displacement amount, and the displacement speed.

In some embodiments, if the required user operation is “bringingtogether two display objects,” the motion determination unit 306 isconfigured to acquire, among the pieces of operation input informationof the output data, the direction of the pinch-in operation (or swipeoperation) to the display, the displacement amount, and the displacementspeed. Then, the motion determination unit 306 is configured todetermine the motion to be performed by the display object (coupling thetwo display objects into one) based on the direction of the pinch-inoperation to the display, the displacement amount, and the displacementspeed. In some embodiments, the motion determination unit 306 isconfigured to determine the direction to move the display object and thespeed of the movement depending on the direction of the pinch-inoperation performed by the user to the display, the displacement amount,and the displacement speed.

In some embodiments, if the required user operation is “operating asteering wheel,” the motion determination unit 306 is configured toacquire, among the pieces of motion input information of the outputdata, the acceleration and the angular velocity. “Operating a steeringwheel” requires fine operations, and hence the motion determination unitcan increase the number of times to acquire the acceleration and theangular velocity within a fixed time period. Then, the motiondetermination unit 306 is configured to determine the motion to beperformed by the display object (changing the direction of the displayobject) based on the acceleration and the angular velocity.

In some embodiments, if the required user operation is “swingingaround”, the motion determination unit 306 is configured to acquire,among the pieces of motion input information of the output data, theacceleration and the angular velocity. If a rough detection issufficient for the operation of “swinging around,” the motiondetermination unit 306 is configured to decrease the number of times toacquire the acceleration and the angular velocity within a fixed timeperiod. Then, the motion determination unit 306 is configured todetermine the motion to be performed by the display object (moving orswinging around the display object) based on the acceleration and theangular velocity. In some embodiments, the motion determination unit 306is configured to determine the strength and direction to swing aroundthe display object based on the acceleration and the angular velocity.

In some embodiments, if the required user operation is “throwing,” themotion determination unit 306 is configured to acquire, among the piecesof motion input information of the output data, the acceleration and theangular velocity. Then, the motion determination unit 306 is configuredto determine the motion to be performed by the display object (throwingthe display object) based on the acceleration and the angular velocity.In some embodiments, the motion determination unit 306 is configured todetermine the strength and direction to throw the display object basedon the acceleration and the angular velocity.

In some embodiments, the required user operation is “speaking out loud,”the motion determination unit 306 is configured to acquire voice inputinformation among the pieces of output data. Then, the motiondetermination unit 306 is configured to determine the motion to beperformed by the display object (enlarging the display object) dependingon the volume. In some embodiments, the motion determination unit 306 isconfigured to determine the size to enlarge the display object to basedon the volume.

The display calculation unit 308 is configured to perform a displaycalculation of the display object based on the motion determined by themotion determination unit 306 and the operation data for the displayobject associated with the motion, to generate an image of the displayobject. The operation data includes acceleration, an angular velocity, adisplacement amount, a displacement speed, and a direction when thedisplay object is moved. When a so-called three-dimensional image isgenerated, display parameters including data of respective apexes of thedisplay object (position coordinates of the apexes in the worldcoordinate system, color data) are calculated, and the images of thedisplay object are sequentially generated for the respective framesbased on the calculated display parameters.

The image generation/output unit 310 is configured to generate, for eachframe, an image in which the calculated display object is placed on ascreen displayed on the display of the activated head-mounted device,and outputs the image to the screen. The HMD 140 is configured todisplay the image output from the image generation/output unit 310.

FIG. 4 is an example of a screen 400 to be displayed on the HMD 140 inaccordance with some embodiments. Screen 400 is generated based on theswipe operation performed by the user. When the output data acquired bythe input detection unit is a swipe operation, the display object beingan object to be operated (six images in FIG. 4) is scrolled in theswiped direction. As illustrated in FIG. 4, indications (402, 404)corresponding to the sensor display operated by the user are virtuallydisplayed on the HMD 140. In this manner, even when the user wearing theHMD 140 is completely immersed in the virtual space, the user canreliably recognize the operation performed or intended to be performedby the user.

FIG. 5A and FIG. 5B are examples of a screen (500A, 500B) to bedisplayed on the HMD 140 in accordance with some embodiments. Screens500A and 500B are generated based on the tap operation performed by theuser. When the output data received from the input detection unit 304corresponds to the tap operation, the display object being the object tobe operated (a hand 502 in FIG. 5A, and a gun 504 in FIG. 5B) performs amotion of punching/shooting. For example, every time the user taps thesensor display 108 of the controller 100, the display object performsthe motion of punching/shooting. Further, when the output data includesthe number of times of tapping within a fixed time period, the speed forthe object to perform the motion of punching/shooting can be changedbased on the number of times of tapping.

FIG. 6 is an example of a screen 600 to be displayed on the HMD 140 inaccordance with some embodiments. Screen 600 is generated based on thetap operation performed by the user. The display screen 600 of FIG. 6includes a sensor display 602 virtually displayed on the virtual space.The virtually-displayed sensor display 602 can virtually display afinger 604 (operation finger) of the user. The user immersed in thevirtual space is wearing the HMD 140, and is disconnected frominformation of the external world. Therefore, the user cannot recognizethe controller 100 present in the real space in detail. By virtuallydisplaying the sensor display of the controller 100 present in the realspace on the virtual space, the user can recognize various operations tobe performed to the controller without taking off the HMD 140.

FIG. 7A, FIG. 7B, and FIG. 7C are examples of screen 700 to be displayedon the HMD 140 in accordance with some embodiments. Screen 700 isgenerated based on the pinch-in/pinch-out operation performed by theuser. When the output data received from the input detection unit 304corresponds to a pinch-out operation, the display object being theobject to be operated (image inside the black circle in FIG. 7A) issubjected to a zoom-in motion of the display object associated with theoperation input. Further, when the output data received from the inputdetection unit includes the displacement amount of the operation, thescreen operation amount of the zoom-in motion can be determined based onthe displacement amount. Further, when the type of the output datareceived from the input detection unit is a pinch-out operation, and thepinch-out operation is performed on the display object being the objectto be operated selected by the user (in FIG. 7B, the center circle withthe number 4), the display object is separated into two objects (in FIG.7B, water droplets) (the object is torn into two pieces). The separationof the object can be also achieved by a swipe operation. When the outputdata received from the input detection unit is a pinch-in operation, thedisplay objects being the objects to be operated (in FIG. 7C, twocircles with the number 2) can be coupled, to thereby obtain one largecircle with the number 4 (two objects are brought together).

FIG. 8A, FIG. 8B, and FIG. 8C are examples of a screen 800 to bedisplayed on the HMD 140, in accordance with some embodiments. When theoutput data received from the input detection unit 304 corresponds tothe user's motion input information obtained from the accelerometer orthe gyroscope, the display object being the object to be operated (asword in FIG. 8A, a fishing rod in FIG. 8B, and a dart in FIG. 8C) issubjected to a motion of swinging around the sword in FIG. 8A, a motionof casting the fishing rod in FIG. 8B, or a motion of throwing the dartin FIG. 8C. Further, the display object can be moved at a predeterminedspeed or acceleration, or the display object can be swung, casted, orthrown at a predetermined force based on the acceleration included inthe output data received from the input detection unit. When the outputdata received from the input detection unit includes the angularvelocity, the motion determination unit can further move the displayobject in a predetermined direction.

FIG. 9A, FIG. 9B, and FIG. 9C are examples of a screen 900 to bedisplayed on the HMD 140 based on the input from the accelerometer orthe gyroscope, in accordance with some embodiments. When the output datareceived from the input detection unit 304 corresponds to the user'smotion input information obtained from the accelerometer or thegyroscope, the display object being the object to be operated (theentire race screen background in FIG. 9A, a person or the whole screenin FIG. 9B, and a door knob in FIG. 9C) is subjected to a motion ofrotating the display object associated with the motion inputinformation. Further, the display object can be rotated at apredetermined angular velocity based on the angular velocity included inthe output data received from the input detection unit.

FIG. 10A and FIG. 10B are examples of a screen 1000 to be displayed onthe HMD 140 based on the input from the microphone sensor in accordancewith some embodiments. When the type of the output data received fromthe input detection unit is a user's voice input obtained from themicrophone, the display object associated with the voice input (a cat inFIG. 10A, and letters of “Ahhhh!!” in FIG. 10B) is subjected to apredetermined motion (motion of obtaining a surprised look of the cat inFIG. 10A, and motion of enlarging the letters of “Ahhhh!!” in FIG. 10B).Further, the size and the speed of the display object to be displayed inan enlarged manner can be changed based on the volume and the directionobtained from the microphone sensor.

FIG. 11 is a flow chart of a process to be performed by the controller100 in accordance with some embodiments.

In step S1100, the interface program 106A is downloaded and installed tothe controller 100 by the antenna of the controller 100 via theInternet. By installing the interface program 106A to the controller100, even when the controller 100 is not a controller dedicated to themain body device 120.

In step S1102, the operation unit 202 acquires the input informationrelating to the predetermined motion/operation performed by the user tothe controller 100, and stores the information to the memory.

In step S1104, the transmission unit 204 transmits, to the main bodydevice 120, as the output data, various types of input relating to thepredetermined motion, various types of input information relating to thepredetermined operation, and/or images or moving images stored in thememory of the controller 100, and the position information, orcombinations thereof. In response to the request from the inputdetection unit 304 of the main body device 120, a transmission unit 204can transmit, among the pieces of output data, the output datacorresponding to the required user operation based on the content of theapplication being executed.

FIG. 12 is a flow chart of a process to be performed by the main bodydevice 120, in accordance with some embodiments.

In step S1200, the main body device-side interface program 126A isinstalled in the main body device 120. The main body device 120 mayconnect to the external network to acquire the interface program 126A,or may use a storage media to acquire the interface program 126A.

In step S1202, the operation determination unit 302 determines therequired user operation that the application allows the user performingoperation to input, based on the content of the application beingexecuted or the like in the main body device 120.

In step S1204, the input detection unit 304 receives and detects, amongthe pieces of output data transmitted from the transmission unit 204 ofthe controller 100, the output data corresponding to the user operationrequired by the application being executed, from the transmission unit204 of the controller 100 via the network 180.

In step 1206, the motion determination unit 306 determines the motion tobe performed by the display object in the virtual space of theapplication being executed, depending on the output data correspondingto the required user operation.

In step S1208, the display calculation unit 308 performs a displaycalculation of the display object based on the motion determined by themotion determination unit 306 and the operation data for the displayobject associated with the motion, to thereby generate the image of thedisplay object.

In step S1210, the image generation/output unit 310 generates, for eachframe, an image in which the generated display object is placed on thescreen displayed on the display of the activated HMD 140, to therebyoutput the image to the HMD 140.

FIG. 13 is a flow chart of a process of connecting the controller 100and the main body device 120 to each other on an application, inaccordance with some embodiments. The controller 100 and the main bodydevice 120 are connected to each other via the network 180 in a mannerthat communication is enabled therebetween.

In step S1300, the main body device 120 activates an application forconnecting to the HMD 140 and operating the HMD 140.

In step S1302, the controller 100 executes the interface program 106Asuch that the controller 100 searches for an application including theinterface program 126A stored in the main body device 120.

In step S1304, when the application including the interface program 126Ais found in the main body device 120, the controller 100 transmits therecognition information of the search to the interface program 126A ofthe main body device 100.

In step S1306, the controller 100 displays the recognized applicationincluding the interface program 126A on the display 108. Then, when theuser taps the displayed recognized application, the controller 100transmits a pairing request to the main body device-side interfaceprogram 126A.

In step S1308, the main body device-side interface program 126Atransmits, to the interface program 106A of the controller 100, apairing response in response to the pairing request transmitted from thecontroller 100.

In step S1310, the interface program 106A of the controller 100transmits, to the main body device-side interface program 126A,information indicating that pairing between the controller 100 and themain body device 120 has succeeded. With this, the connection betweenthe controller 100 and the main body device 120 on the application iscompleted.

FIG. 14 is a block diagram of a system comprising of a plurality ofcontrollers, in accordance with some embodiments. In FIG. 14, theplurality of controllers 100 are connected to each other not via themain body device 120 but via the Internet. A first controller 100A and asecond controller 100B can each include a processor, a memory, aninterface, and a sensor display as discussed with respect to controller100 shown in FIG. 1. In some embodiments, the first controller 100A andthe second controller 100B each include the functional configurationillustrated in FIG. 2, or a part of the functional configurationillustrated in FIG. 2. In some embodiments, the plurality of controllerscomprises three or more controllers.

In FIG. 14, only one controller 100A is connected to one main bodydevice 120, and the other controllers 100B are connected to the server160 via the Internet. With this configuration, without increasing theload on the one main body device 120, various operations can beperformed on the one operation screen by a plurality of users withoutimposing a high load on system resources. Further, the plurality ofusers are not required to be located in the same space, and can belocated in different spaces.

FIG. 15 is a diagram of a user interface in which one operation screenis operated by a plurality of users, in accordance with someembodiments. In FIG. 15, a user A is wearing the HMD 140, but a user Bis not wearing an HMD. The user B not wearing the HMD can also view theoperation screen of the user A via the display of the second controller100B, or can perform, with respect to the screen being operated by theuser A, an operation input of swiping, tapping, or pinching with use ofthe sensor display of the second controller 100B, or motion input withuse of the accelerometer and the angular velocity sensor of the secondcontroller 100B. For example, as illustrated in FIG. 15, while the userA is playing a racing game, another user B can use the sensor display ofthe second controller 100B of the user B to place an obstacle (banana inFIG. 15) in the screen being operated by the user A.

An aspect of this description is related to a system comprising a firstdevice configured to cause a virtual reality image to be output by adisplay. The system also comprises a second device communicativelycoupled with the first device. The second device is configured to detecta user input associated with an action capable of being displayed in thevirtual reality image. The second device is also configured to cause thefirst device to cause an object to be displayed in the virtual realityimage based on the user input. The object at least partially comprises agraphic representative of one or more of the second device, a usermovement with respect to the second device, or the user input.

Another aspect of this description is related to a method comprisingcausing, by a first device, a virtual reality image to be output by adisplay. The method also comprises detecting, by a second devicecommunicatively coupled with the first device, a user input associatedwith an action capable of being displayed in the virtual reality image.The method further comprises causing, by the second device, the firstdevice to cause an object to be displayed in the virtual reality imagebased on the user input. The object at least partially comprises agraphic representative of one or more of the second device, a usermovement with respect to the second device, or the user input.

A further aspect of this description is related to an apparatuscomprising at least one processor, and at least one memory connected tothe at least one processor and including computer program code for oneor more programs. The at least one memory and the computer program codeare configured to, with the at least one processor, cause the apparatusto cause a virtual reality image to be output by a display based onreceived data indicative of a user input associated with an actioncapable of being displayed in the virtual reality image. The apparatusis also caused to cause an object to be displayed in the virtual realityimage based on the data indicative of the user input. The object atleast partially comprises a graphic representative of one or more of thesecond device, a user movement with respect to the second device, or theuser input.

The above-mentioned embodiments are merely an example for facilitatingan understanding of the present disclosure, and does not serve to limitan interpretation of the present disclosure. It is to be understood thatthe present disclosure can be changed and modified without departingfrom the gist of the disclosure, and that the present disclosureincludes equivalents thereof.

What is claimed is:
 1. A system, comprising: a first device configuredto cause a virtual reality image to be output by a display; and a seconddevice communicatively coupled with the first device, the second devicebeing configured to: detect a user input associated with an actioncapable of being displayed in the virtual reality image; and instructthe first device to cause an object to be displayed in the virtualreality image based on the user input, wherein the object at leastpartially comprises a graphic representative of one or more of thesecond device, a user movement with respect to the second device, or theuser input.
 2. The system according to claim 1, wherein the first devicecomprises the display.
 3. The system according to claim 1, wherein thefirst device is communicatively coupled with the display.
 4. The systemaccording to claim 3, wherein the display is a head mounted display. 5.The system according to claim 1, wherein the second device is configuredto have connectivity to a communication network independent of the firstdevice.
 6. The system according to claim 1, wherein at least one of thefirst device or the second device is a mobile device.
 7. The systemaccording to claim 6, wherein the user input comprises one or more of aswipe operation, a tap operation, and a pinch-in/pinch-out operation. 8.The system according to claim 1, wherein the user input is based on avoice input performed by the user, and the object comprise one or moreof an image corresponding to the voice input or a textual representationof the voice input.
 9. The system according to claim 8, wherein seconddevice is further configured to cause the object displayed by thevirtual space to change based on the voice input.
 10. The systemaccording to claim 1, wherein the user input is based on a motion inputperformed by the user, and the second device comprises one or more of anaccelerometer, a gyroscope or a GPS unit configured to output data uponwhich the motion input is based.
 11. The system according to claim 1,wherein the graphic comprises at least a finger of a user positioned inthe virtual reality image with respect to an image of the second device.12. The system according to claim 1, wherein the second device isconfigured to cause the object to move in the virtual reality imagebased on the user input.
 13. The system according to claim 1, whereinthe user input is a first user input, the action is a first action, theobject is a first object, and the system further comprises: a thirddevice communicatively coupled with the second device by way of acommunication network independent of the first device, the third devicebeing configured to: detect a second user input associated with a secondaction capable of being displayed in the virtual reality image; andcause the first device, by way of the second device, to cause a secondobject to be displayed in the virtual reality image based on the seconduser input, wherein the second object comprises an image associated withthe second action or at least partially comprises a graphicrepresentative of one or more of the third device, a user movement withrespect to the third device, or the second user input.
 14. The systemaccording to claim 13, wherein the display is a first display, and thethird device is further configured to cause a second display to outputthe virtual reality image, and at least one of the first object or thesecond object.
 15. The system according to claim 14, wherein the thirddevice comprises the second display.
 16. The system according to claim1, wherein the display is a first display, and the system furthercomprises: a third device communicatively coupled with the second deviceby way of a communication network independent of the first device, thethird device being configured to cause a second display to output thevirtual reality image.
 17. The system according to claim 16, wherein thethird device is further configured to cause the second display to outputthe object in the virtual reality image output by the second display.18. The system according to claim 1, further comprising: a server; and athird device communicatively coupled with the server by way of acommunication network independent of the first device, wherein thesecond device is communicatively coupled with the server by way of thecommunication network independent of the first device, the display is afirst display, and the third device is configured to cause a seconddisplay to output the virtual reality image based on data received fromthe server.
 19. A method, comprising: causing, by a first device, avirtual reality image to be output by a display; detecting, by a seconddevice communicatively coupled with the first device, a user inputassociated with an action capable of being displayed in the virtualreality image; and causing, by the second device, the first device tocause an object to be displayed in the virtual reality image based onthe user input, wherein the object at least partially comprises agraphic representative of one or more of the second device, a usermovement with respect to the second device, or the user input.
 20. Anapparatus, comprising: at least one processor; and at least one memoryconnected to the at least one processor and including computer programcode for one or more programs, the at least one memory and the computerprogram code are configured to, with the at least one processor, causethe apparatus to: cause a virtual reality image to be output by adisplay based on received data indicative of a user input associatedwith an action capable of being displayed in the virtual reality image;and cause an object to be displayed in the virtual reality image basedon the data indicative of the user input, wherein the object at leastpartially comprises a graphic representative of one or more of thesecond device, a user movement with respect to the second device, or theuser input.